Model-free deepreinforcement learning (RL)algorithms havebeenwidely used for a range of complex control tasks.

Finally, we extend Min-AM to solve stochastic programming problems.

In this paper, we provide deeper insights into Anderson acceleration in reinforcement learning by establishing its connection with quasi-Newton methods for policy iteration and improved convergence guarantees under the assumptions that the Bellman operator is differential and non-expansive.

Neural Information Processing Systems http://nips.cc/

Our algorithm is able to solve problems with millions of samples and features in seconds, by relying on coordinate descent, working sets and Anderson acceleration.

One major concern of AM is the heavier memory overhead compared with the fixed-point iteration.

Federated learning (FL) is a distributed machine learning approach that enables multiple local clients and a central server to collaboratively train a model while keeping the data on their own devices. First-order methods, particularly those incorporating variance reduction techniques, are the most widely used FL algorithms due to their simple implementation and stable performance. However, these methods tend to be slow and require a large number of communication rounds to reach the global minimizer. We propose FedOSAA, a novel approach that preserves the simplicity of first-order methods while achieving the rapid convergence typically associated with second-order methods. Our approach applies one Anderson acceleration (AA) step following classical local updates based on first-order methods with variance reduction, such as FedSVRG and SCAFFOLD, during local training. This AA step is able to leverage curvature information from the history points and gives a new update that approximates the Newton-GMRES direction, thereby significantly improving the convergence. We establish a local linear convergence rate to the global minimizer of FedOSAA for smooth and strongly convex loss functions. Numerical comparisons show that FedOSAA substantially improves the communication and computation efficiency of the original first-order methods, achieving performance comparable to second-order methods like GIANT.

The main contribution of this paper is to apply Anderson acceleration to the setting of deep reinforcement learning. The authors first propose a regularized form of Anderson acceleration, and then show how it can be applied to two practical deep RL algorithms: DQN and TD3. Originality: This paper falls under the vein of applying existing techniques to a novel domain. While the idea of introducing Anderson acceleration to the context of RL is not new, as the authors mention, it has not been applied to deep RL methods. While the originality is somewhat limited in this aspect, developing a practical and functional improvement for deep RL algorithms is not trivial.